Visible and infrared self-supervised fusion trained on a single example

This paper addresses the problem of visible (RGB) to Near-Infrared (NIR) image fusion. Multispectral imaging is an important task relevant to image processing and computer vision, even more, since the development of the RGBT sensor. While the visible image sees color and suffers from noise, haze, and clouds, the NIR channel captures a clearer picture and it is significantly required by applications such as dehazing or object detection. The proposed approach fuses these two aligned channels by training a Convolutional-Neural-Network (CNN) by a Self-Supervised-Learning (SSL) on a single example. For each such pair, RGB and IR, the network is trained for seconds to deduce the final fusion. The SSL is based on Sturcture-of-Similarity (SSIM) loss combined with Edge-Preservation (EP) loss. The labels for the SSL are the input channels themselves. This fusion preserves the relevant detail of each spectral channel while not based on a heavy training process. In the experiments section, the proposed approach achieves better qualitative and quantitative multispectral fusion results with respect to other recent methods, that are not based on large dataset training

Fast Detection of Curved Edges at Low SNR

Detecting edges is a fundamental problem in computer vision with many applications, some involving very noisy images. While most edge detection methods are fast, they perform well only on relatively clean images. Indeed, edges in such images can be reliably detected using only local filters. Detecting faint edges under high levels of noise cannot be done locally at the individual pixel level, and requires more sophisticated global processing. Unfortunately, existing methods that achieve this goal are quite slow. In this paper we develop a novel multiscale method to detect curved edges in noisy images. While our algorithm searches for edges over a huge set of candidate curves, it does so in a practical runtime, nearly linear in the total number of image pixels. As we demonstrate experimentally, our algorithm is orders of magnitude faster than previous methods designed to deal with high noise levels. Nevertheless, it obtains comparable, if not better, edge detection quality on a variety of challenging noisy images.Comment: 9 pages, 11 figure

Registration and Fusion of Multi-Spectral Images Using a Novel Edge Descriptor

In this paper we introduce a fully end-to-end approach for multi-spectral image registration and fusion. Our method for fusion combines images from different spectral channels into a single fused image by different approaches for low and high frequency signals. A prerequisite of fusion is a stage of geometric alignment between the spectral bands, commonly referred to as registration. Unfortunately, common methods for image registration of a single spectral channel do not yield reasonable results on images from different modalities. For that end, we introduce a new algorithm for multi-spectral image registration, based on a novel edge descriptor of feature points. Our method achieves an accurate alignment of a level that allows us to further fuse the images. As our experiments show, we produce a high quality of multi-spectral image registration and fusion under many challenging scenarios

Classic versus deep learning approaches to address computer vision challenges : a study of faint edge detection and multispectral image registration

Computer Vision involves many challenging problems. While early work utilized classic methods, in recent years solutions have often relied on deep neural networks. In this study, we explore those two classes of methods through two applications that are at the limit of the ability of current computer vision algorithms, i.e., faint edge detection and multispectral image registration. We show that the detection of edges at a low signal-to-noise ratio is a demanding task with proven lower bounds. The introduced method processes straight and curved edges in nearly linear complexity. Moreover, performance is of high quality on noisy simulations, boundary datasets, and real images. However, in order to improve accuracy and runtime, a deep solution was also explored. It utilizes a multiscale neural network for the detection of edges in binary images using edge preservation loss. The second group of work that is considered in this study addresses multispectral image alignment. Since multispectral fusion is particularly informative, robust image alignment algorithms are required. However, as this cannot be carried out by single-channel registration methods, we propose a traditional approach that relies on a novel edge descriptor using a feature-based registration scheme. Experiments demonstrate that, although it is able to align a wide field of spectral channels, it lacks robustness to deal with every geometric transformation. To that end, we developed a deep approach for such alignment. Contrarily to the previously suggested edge descriptor, our deep approach uses an invariant representation for spectral patches via metric learning that can be seen as a teacher-student method. All those pieces of work are reported in five published papers with state-of-the-art experimental results and proven theory. As a whole, this research reveals that, while traditional methods are rooted in theoretical principles and are robust to a wide field of images, deep approaches are faster to run and achieve better performance if, not only sufficient training data are available, but also they are of the same image type as the data on which they are applied

Deep Multi-Spectral Registration Using Invariant Descriptor Learning

In this paper, we introduce a novel deep-learning method to align cross-spectral images. Our approach relies on a learned descriptor which is invariant to different spectra. Multi-modal images of the same scene capture different signals and therefore their registration is challenging and it is not solved by classic approaches. To that end, we developed a feature-based approach that solves the visible (VIS) to Near-Infra-Red (NIR) registration problem. Our algorithm detects corners by Harris and matches them by a patch-metric learned on top of CIFAR-10 network descriptor. As our experiments demonstrate we achieve a high-quality alignment of cross-spectral images with a sub-pixel accuracy. Comparing to other existing methods, our approach is more accurate in the task of VIS to NIR registration

Multispectral image fusion by super pixel statistics

Multispectral image fusion is a fundamental problem of image processing and remote sensing. This problem is addressed by both classic and deep learning approaches. This paper is focused on the classic solutions that can work in real-time systems and introduces a new novel approach to this group of works. The proposed method carries out multispectral image fusion based on the content of the fused images. Furthermore, it relies on an analysis of the level of information of segmented superpixels in the fused inputs. Specifically, the proposed method addresses the task of visible color RGB to Near-Infrared (NIR) fusion. The RGB image captures the color of the scene while the NIR channel captures details and sees beyond haze and clouds. Since each channel senses different information of the scene, their multispectral fusion is challenging and interesting. Therefore, the proposed method is designed to produce a fusion that contains the relevant content of each spectra. The experiments of this manuscript show that the proposed method is visually informative with respect to other classic fusion methods. Moreover, it can be run fastly on embedded devices without heavy computation requirements